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Abstract 



Five simple guidelines are proposed to compute the generating function for the nonnegative 
integer solutions of a system of linear inequalities. In contrast to other approaches, the emphasis 
is on deriving recurrences. We show how to use the guidelines strategically to solve some 
nontrivial enumeration problems in the theory of partitions and compositions. This includes a 
strikingly different approach to lecture hall- type theorems, with new g-series identities arising in 
the process. For completeness, we prove that the guidelines suffice to find the generating function 
for any system of homogeneous linear inequalities with integer coefficients. The guidelines can be 
viewed as a simplification of MacMahon's partition analysis with ideas from matrix techiniques, 
Elliott reduction, and "adding a slice" . 

1 Introduction 

This continues our work in [181 119| studying nonnegative integer solutions to linear inequalities 
as they relate to the enumeration of integer partitions and compositions. Define the weight of a 
sequence A = (Ai, A2, . . . , A„) of integers to be |A| = Ai + ■ • • + A„. If sequence A of weight N has all 
parts nonnegative, we call it a composition of N; if, in addition, A is a nonincreasing sequence, we 
call it a partition of N. 

Given an r x n integer matrix C = [ci_j], we consider the set Sc of nonnegative integer sequences 
A — (Ai, A2, . . . , A„) satisfying the constraints 



Ct,0 + Ci,lAi + Ci^2A2 + . 

We seek the full generating function 



+ Q,„A„ > 0, 



Ki <r. 



(1) 




(2) 



AeSc 
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which can be viewed as an encapsulation of the solution set Sc- the coefficient of in Fc{qxi, qx2, ■ ■ ■ , qxn) 
is a listing (as the terms of a polynomial) of all nonnegative integer solutions to of weight N and 
the number of such solutions is the coefficient of q^ in Fc{q, q, . . . ,q). 

Variations of this problem arise in other areas of mathematics, e.g., solving systems of linear equa- 
tions, finding volume of polytopes, as well as in enumeration. In the papers |18[I19| we demonstrated 
that in the area of partition and composition enumeration many familiar sets of linear constraints 
can be easily handled a matrix inversion: for homogeneous systems, if the constraint matrix C is 
an n X rt invertible matrix, and if all entries of C^^ ^ B = [bij] are nonnegative integers then by 
Theorem 1 in jl9| : 

n ^ 
Fc{x,,X2, . . . ,x„) = Y[ _ — 

j=l -''1 • • ■ Xn ) 

This theorem (in its full generality) suffices to handle the enumeration of such families as Hickerson 
partitions 22, , Santos' interpretation of Euler's family ,28, , Sellers' generalization of Santos |29ll30j . 
partitions with nonnegative second differences 0, super-concave partitions 'Si , partitions with 
r-th differences nonnegative [3i il4. ,33^ . partitions with mixed difference conditions and examples 
(0-5) of Pak in 27 . The theorem provides bijections as well as generating functions. 

However, it is easy to find simple examples where the "C matrix" technique fails. In Section 2, 
we propose five simple guidelines for computing the generating function Fq of a system C of linear 
diophantine inequalities. The guidelines can be viewed as a simplification of MacMahon's partition 
analysis with ideas from matrix methods, Elliott reduction [21], and "adding a shce" (e.g. P5)- 

Our focus is on the use of the guidelines to derive a recurrence for the generating function Fc^ 
of an infinite family {C„|n > 1} of constraint systems. This is in contrast to the focus of the Omega 
package a software implementation of partition analysis, well-designed to compute the generating 
function of a given fixed, finite system of linear constraints. The advantage of a recurrence for Fc„ 
is a program which computes i^c„ for any given n. But more significantly, if the recurrence can be 
solved, it provides a closed form for the generating function for the infinite family. 

In Sections 3-6, we show how to use the guidelines of Section 2 strategically to solve some 
nontrivial enumeration problems in the theory of partitions and compositions. Sections 3 and 4 
address well-studied problems, included as "warm-up" exercises to illustrate the approach and the 
handling of the recurrences that result. Sections 5 and 6 apply the method to the problem of 
enumerating anti-lecture hall compositions |lf)j and truncated lecture hall partitions |17| . giving a 
simpler approach than in ^JEI- For completeness, in Section 7 we prove that the guidelines suffice 
to find the generating function for the nonnegative integer solutions of any homogeneous system of 
linear inequalities with integer coefficients. 

This work was inspired by the the work of Andrews, Paule, and Riese in the sequence of papers 
OElElllliniCllHlinilSllinillll, which mustrate many applications of partition analysis. The Omega 
Package software was an invaluable tool in our early investigations. As illustrated in papers such 
as 2, 3, l^lini lTnj . recurrences can certainly be derived using partition analysis. However, we found 
that the task became easier with a simpler set of tools which appear to be no less powerful. In 
Section 8 we discuss MacMahon's partition analysis and show how the proposed guidelines can be 
be viewed as essential ideas underlying his theory. 
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2 The Five Guidelines 



Let C be a set of linear constraints in n variables, Ai, . . . , A^, each constraint c € C of the form 

n 

c: [ao + y^^QiXj > 0], 

i=l 

for integer values oq, ai, . . . , a„. 

Let Sc be the set of nonnegative integer sequences A = (Ai, . . . , A„) satisfying all constraints in 
C. Since we are only interested here in nonnegative integer solutions, we will always assume that C 
contains the constraints [Aj > 0] for 1 < i <n. Define the full generating function of C to be: 

AeSc 

If c is the constraint: [ao + OjA, > 0] define the negation of c, -ic, to be the constraint 

[—ao — X^iLi osjAj > 1]. Then any sequence (Ai, . . . , A„) satisfies c or -ic, but not both. A constraint 
c is implied by the set of constraints C if Scu{^c} = 0- A constraint c is redundant if Scu{c} = Sc- 

Let CAi^Ai+aA denote the set of constraints which results from replacing A^ by A^ + aXj in every 
constraint in C. Note that if constraint c is implied by C then CA^^Ai+aAj is implied by Cx.^x^-^-aXJ■ 
Thus observe that if C contains the constraints [Afc > 0] , 1 < < n and if [Aj — aXj > 0] is implied 
by C, then all of the constraints [Afc > 0], 1 < < n are also implied by Cx^^x^+aX-,- 

Lemma 1 Let C be a set of linear constraints on variables Ai , . . . , A„ which contains the constraints 
[Afc > 0], 1 < A; < n. Let a be any integer (possibly negative). Suppose [Aj — aXj > 0] is implied by C 
and let C = Cxi^Xi+aXj ■ Then 

/3 = (/3i,...,/J„)e5c iff P' = {Pi,...,f3i-i,(}i-af3j,pi+i,...,f3n)eSc'. 



Proof. By the remarks preceding the lemma, the constraints C and C guarantee that Sc and Sc 
contain only nonnegative integer solutions. So, it suffices to show that /? satisfies a constraint in C 
iff /?' satisfies the corresponding constraint in C. 

Let c(A) = Co + J27=i '^'^f assume [c(A) > 0] G C. Under the substitution Aj <— Aj + aXj, 
c(A) becomes c'(A) defined by 

n 

c'(A) = Co + ^ ctXt + CiuXj = c(A) + CiuXj 
t=i 

and [c'(A) > 0] e C. Thus 

c{/3)=c\P)-Ciaf3j = c'{(3'), 
so c(/3) > iff c'(/3') > 0. □ 

Finally, to simplify notation, we will let X„ refer to the parameter list Xi,. . . , Xn, so that F{Xn) 
denotes F{x-i, . . . ,a;„). Let -F(X„; ) denote the function F{Xn) with all occurrences of Xi 

replaced by XiXj. 
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Theorem 1 (The Five Guidelines) 

1. If C = {[Ai > t]}, for integer t > 0, then 

Fc{xi) = 



1 — Xi 



2. If Ci is a set of constraints on variables Ai, . . . , Aj and C2 is a set of constraints on variables 
. . . , A„, then 

FciuC2{^i, ■ ■ -^^n) = Fciixi, ■ ..Xj)Fc:,{xj+i, . . . ,a;„). 

3. Let C be a set of linear constraints on variables Ai, . . . , A„ and assume C contains the con- 
straints [Ai > 0], 1 < i < n. Let a be any integer (possibly negative). If [Xi — aXj > 0] is implied by 
C, 

Fc{Xn) = ^c.^^.,+„.^ (X„; X, <- 
4-. Let c be any constraint with the same variables as the set C. Then 

5. Let c £ C. Then 

Fc{Xn) = Fc-lc}{Xn) - Fc-{c}ii{^c}{X„). 



Proof. 

1. This is clear since Fc{xi) = x\ + x\'^^ + ■ • • . 

2. The sequence (Ai, . . . , A„) G S'ciuCs iff (-^i: ■ • ■ , Aj) 6 and (A^+i, . . . , A„) G Sc2- 

3. Let C = CAi^Ai+aAj - By Lemman] 

(Ai, . . . , A„) e 5'c' iff (Ai, . . . , Ai_i, Ai + aAj, Ai+i, . . . , A„) e S'c- 



So, 



Fc'{Xn',Xj< XjX^) — ^ ' X2^ ■ ■ ■ Xj^-^ [XjXi) 'Xji^-^ ■ ■ ■ T. 



^ « J .... , J , ^ '^n 

xesc 

■^1 X2 -^i-l X^ x„ 



. . . . . . ™A 



AeSc 

4. Sc can be partitioned into those A that satisfy c and those that do not. 

5. By guidehne 4, i^c-{c}(^n) = i^c-{c}u{c}(^«) + i^c-{c}u{^c}(^«)- Then C - {c} U {c} = C, 
since c G C. □ 
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3 Mine's Partition Function and Cayley Compositions 



Mine's partition function ^{d, N) is the number of compositions of N in which the first part is d and 
each part is at most twice the size of the preceding part j26) . For example, in the special case d = 1, 
these are called Cayley compositions ^mEl- In this section we compute the generating function 

= J2d,N>o ^('^^ =q + 2q^+ 4g3 + 7^4 _^ 13^5 _^ 2Aq^ H . For example, the coefficient of 

is 13, since of the 16 compositions of 5, only these three violate the constraints: (1,4), (1,3,1), 
and (1,1,3). 

Let Cn be the set of constraints C„ = {A^ > ^A^+i > | 1 < i < 71} and let C„(xi, . . . ,a:„) 
be the generating function of C„. Focusing on the constraint c = [A„_i > ^A„], after noting that 
[A„_i > 0] is redundant, we can write C„ as 



Ai 


> 


\\2 


A2 


> 


^Aa 


n-2 


> 


1 \ 


71 — 1 


> 


1 \ 


An 


> 






Ai 


> 




A2 


> 


5A3 


n-2 


> 


1 \ 
2^" 


n — 1 


> 


1 \ 


n — 1 


> 





An 


> 






Ai 


> 


iA2 




" Ai 


> 


iA2 


A2 


> 


|A3 




A2 


> 


I A3 


n-2 


> 


|An-l 




An -2 


> 


|A„-i 


n— 1 


> 







An 


> 


2An-l 


An 


> 







. A„_i 


> 






where c has been removed from the next-to-last system, making it C„-i U [A„ > 0], and c has been 
replaced by in the last system. By guidelines 1 and 2, a;„C„_i(a;i, . . . ,x„_i)/(l — Xn) is the 
generating function for C„_i U [A„ > 0]. Note further that the substitution A„ ^ A„ + 2A„-i in 
the last system results C„-i U [A„ > 0], so by guideline 3, the last system has generating function 
XnCn-i{xi, . . . ,Xn-ix'^)/{l — Xn)- Putting this together with guideline 5 and the initial condition 
Ci{xi) = Xi/{1 — xi) gives the recurrence 



Cn {xi , . . . , Xn) 



-(C„_l(xi, . . . ,Xn-l) - C„_l(xi 



, Xn—2: Xn — lX^^)). 



Let Cn{q, s) = C„(g, q, . . . ,q, s). Then the above recurrence gives Ci(g, s) = s/(l — s) and for n > 2, 



-{Cn-i{q,q) - Cn-i{q,qs'^)). 



Cn{q, s) 



Set C{q, s) — J2'^=i Cn{q, s) and use the recurrence for C„(<;, s) to get 



Ciq,s) = ^C„((Z,s) = ^^+Y^Cniq,s) = ^^{l + C{q,q)-Ciq,qs')). 
n=l ' n=2 



Iterating the recurrence for C{q,s) gives 

C{q,s) = {l + Ciq,q))Y,{-l)^-'l[- 

i=l j=0 ^ 

Let C{q) — C{q,q), then 

,.{q) = 1 + C{q) 



q"^ ^s"^ 



i=l 



1 



1 + E.= 17T 



(l_^)(l_g3)(l_,7)...(l_,2.-l) 
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4 Two-Rowed Plane Partitions 



This example illustrates the advantage of guideline 3 of Theorem ^ when a < 0. The two-rowed 
plane partitions are those integer sequences (ai, 6i, . . . , a„, 6„) satisfying the constraints 



Vn = [ai>bi>0, 1 < i < ?i; > a^+i, 

It is well-known that the generating function for Vn is (24] 



bi > bi 



1 < I < 71 - 1] . 



(3) 



In PI, Andrews shows how MacMahon's partition analysis can be used to compute Pn{q) by con- 
sidering an intermediate family Qn- We will use this approach, but with a slight twist, to show how 
the generating function for Vn, can be computed via Qn from the guidelines of Theorem ^ 

We will use the convention that when a constraint system is represented by a calligraphic letter, 
its generating function is represented by the corresponding roman letter. Also, to keep notation 
simple, when the meaning is clear from context, we will use the same letter to refer to multivariable 
and single variable forms of the generating function. 



Define Qn to be the set of constraints below: 



ai + a2 + ■ ■ ■ + a„ 
a2 + ■ ■ ■ + a„ 



Qn 



> 6i + &2 H h b„ 

> fe2 H h b„ 



ffln-l 4- tin ^ 



an 

ai,h > 0, 



> 



bn-1 + bn 
bn 

i = 1, . . . , n 



Denote the full generating functions for Vn and Qn by 

Pn{xi,yi,...,Xn,yn) ^ ^ 

{ai,bi,...,an,bn)(^S-Pn 



ai bi bji 
^1 Ul • ■ • J ^n Vn I 



Gn{Xl , J/l, . . . , Xm Vn) 



E 



ai b 



Vi ■■■,Xn"y, 



{a\,b\,...,an,bn)^Sg^ 

Note that Vn can be transformed into Qn by the sequence of substitutions: 

ai^ai + Oi+i] bi^bi + bi+i; i = 1, 2, . . . n - 1. 



We focus on G„. Since for 1 < i < n — 1, Oi — a^+i > and bi — bi+i > in V, by guideline 3 of 
Theorem n] P„ is obtained from G„ by the sequence of substitutions: 

Xi ^ XiXi^i; y.i'^y^yi^i z = n, n - 1, n - 2, . . . , 2. 

Thus 

. . . ,Xn,yn) = Gn{xi,yi,xiX2,yiy2, ■ • ■ ,xiX2 ■ ■■Xn,yiy2 ■ ■ - Vn)- 
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In particular, the generating function ||2Jl for two-rowed plane partitions is obtained by setting 
Xi =Vi = q in P„ for « = 1, . . . , n: 



Pn{q,q,q,...,q) = Gn{q,q,q ,9 



(4) 



Since a„ — 6n > in C?„, by guideline 3, we can do the substitution a„ ^ a„ + bn in Qn to get Tn 
and recover G„ from Fn as shown below. 



ai + ^2 H hfln > &1 + &2 H +&n-l 

a2 + ■ • • + (1,1 > fe2 + ■ ■ ■ + &n-l 



a,i— 1 + a,i > 
fli, &i > 0, 



i — 1, . . . ,n 



Since a„_i + a„ > in JF„, by guideline 3, we can substitute a„_i <— a,i_i — a„ in Tn to get 7-^„ 
and recover Fn from _ff„ as shown. 

fli + a2 + ■ • • + a„_i > fei + 62 + ■ • • + fen-i 
a2 + ■ • • + a„_i > 62 + ■ • • + 6ri-i 



ai, 6i > 



an 



Summarizing to this point, wc have 

, Ul , . . . , Xji , T/tj^ ) 1 yi 7 ■ ■ ■ 7 *^n — 1 7 Un — l i •^n / *^n — 1 1 •^nyn}- 



(5) 



Now apply guideline 5 to ?i„ using the constraint c = [a„_i > a„]. Then 

7 



where 
that is, 



and 



ICn = 'Hn- {[flri-l > On]}, jC-n = H„ - {[a„_i > a„]} U { [a„ > a„_i + 1]}, 



ai+a2-\ + fl,i-i > bi+b2 



02 



•■ + an-i > 
fli, 6i > 



+ • • ■ + 

+ • ■ ■ + bn-1 
bn-1 

i = 1, . . . ,n 



ai+a2 + ha„-i > 61+62 + h 6„-i 

a2 H h a„-i > 62 H h 6„-i 



a„-i > 
a„ > 



bn-l 

a„_i + 1 



a.,6i > 



i= 1, 
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so that 

Hn{xi,yi, . . . ,Xn,yn) ^ Kn{ ) - Ln{xi,yi,...,Xn,yn)- (6) 

Now observe that 

u{K > 0],[6„ > 0]}, 

so by guidehnes 1 and 2, 

7^ / N Gn-l(,Xi,yi, . . . ,Xn-l,yn-l) /_n 

Kn{Xl,yi,...,Xn,yn) = 7^ T^j T • (7) 

(1 - X„)(l - yn) 

Returning to £„, since a„ — a,i_i > in £„, we can do the substitution a„ ^ a,i + a„_i, resuhing 
in 

U{[a„ > 1],[6„ > 0]}, 

so by guidehnes 1, 2, and 3, 

7- / \ ■^nG'ji — 1 (3^1 , yi , . ■ ■ T Xji—1 : yn — 1 : Xn~l ^ ^n — l-^n) /^\ 
Ln{Xl,yi,- ■ ■ ,Xn,ynj = 7" ry- r . (8) 

(1 - a;„)(l - y„) 

Combining ®,lI7J, and |(SJ, we have 

Gn-l{xi,yi, ■ ■ ■ ,Xn-l,yn-l) a;„G„_i (xi , J/i , . . . , Xn-2,yn-2, Xn-lXn, yn-l) 



Hn{xi,yi, . . . ,Xn,yn) = 



{1 - Xn){l - yn) {l-Xn){l-yn) 



Finally, substituting this expression for _ff„ into (jsj gives a recurrence for G„: 

Gn-i{xi l) - ir^Gn-l{xi,yi, . . . ,Xn~2,yn~2,Xn,yn-l) 



Gn{xi, Hi, ■ ■ ■ ,Xmyn) 



(1 - Xn/Xn-l){l - Xnyn) 



(9) 



with initial condition Gi(xi, yi) = 1/(1 — xi)/ {\ — xiyi). 

Let G'^{q, s) — Gn{q, q,q^ , ■ ■ ■ , s, g"). Then from the recursion JU)), 

G*_i(g,(z"-i)-(s/g"-i)G*_i(g,s) 



(1 - s/9"-i)(l - sq") 



It is straightforward to show by induction that G* (q, s) satisfies 

1 

^"^'^'"^ " (l-s)(l-.s(?)((?;g)„_i(g2;q)„_i- 

Substituting s — g" gives 

Pn{q) = G„(g, <z, g^ ...,q\ g") = G:(g, g") = . , ) ^ , , 
the desired generating function for 2 x n plane partitions. 
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5 Anti-Lecture Hall Compositions 



In JS], we considered the set of sequences A — (Ai, . . . , A^) satisfying the constraints 



Ai A9 

— > — > 
1 - 2 - 



> — >o 

n 



We referred to these as anti-lecture hall compositions and showed that the generating function is 

1 + q' 



(10) 



Here we show how to apply the guidehnes of Theorem^to get a recurrence for the full generating 
function v4„(xi, X2, . . . x„) and use it to give an "easy" proof of ifTUI) . The idea is easily extended to 
the truncated anti-lecture hall compositions studied in '17'. We start with Bm a slight variation of 



Lemma 2 The full generating function for the integer sequences defined by the constraints 

Bn = 



Ai A9 

— > — > 
1 - 2 - 



> hltzL > ^ >o 
- n - 1 - 1 - 



(11) 



satisfies 



An-i{xi . . . ,a:„_i) 



Proof. The following sequence of substitutions transforms Bn into An-i U{[A„ > 0]}, as illustrated 
in Figure n] 

Ai ^ Ai + i\n, i = n - I, . . . ,1. 

Note that the constraint Ai_i > {i — l)\n is implied at each stage, so by guidelines 1,2, and 3, Bn 
is recovered from An by performing the sequence of substitutions on An-i{xi, . . . ,x„_i)/(l — Xn)'- 

Xn ^ XnXj^^ i 1,...,71 1. 



Proposition 1 The full generating function for anti-lecture hall compositions satisfies: 



An{xi, . ..Xn) 



■ ■ ■ ,Xn^l) 
1 Xn 



An — 1 (^1 , ■ ■ ■ , Xn — 2 1 XnXn^l) 



1 



with initial condition A\(x\) = 1/(1 — Xi). 



Proof. Using guideline 5 with c = [A„_i > ^^^A„], 



□ 



-^n (^1 ; ■ ■ ■ ) *^n) — Cn (xi , . . . , Xn ) (^^1 : ■ ■ ■ 5 •^n ) ? 
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Ai 


> 


iA2 


A2 


> 


iA3 


An-3 


> 




An-2 


> 


n — 2 \ 


An-1 


> 


(n - l)An 


An 


> 






Ai 



iA2 



> 2-^2 



A2 > iXs 



An-3 


> 


Ti — 3 \ 


An-2 


> 


f^An-i + (n 


An-1 


> 





An 


> 








" Ai 


> 


iA2 




" Ai 


> 


iA2 + An 




A2 


> 


IA3 + 2 An 




A2 


> 


|A3 




An-3 


> 






An-3 


> 






An-2 


> 


n — 2 \ 




An-2 


> 


n-2 \ 




An-1 


> 







An-1 


> 







An 


> 







An 


> 






An-3 
An-2 
An-1 



> 


IA2 




> 


3 A3 




> 


n — 3 \ 


n-2 + 


> 


n-l'^ 


n — 1 


> 







> 







Ai 


> 


iA2 


A2 


> 


IA3 


An-3 


> 




An-2 


> 


n-2 


An-1 


> 





An 


> 






Figure 1: Transformation of S„ into An-i U {[A„ > 0]} in proof of Lemma |2| 



where 



r — 



Ai A2 
— > — > 
1 - 2 - 



Ai A2 
— > — > 
I - 2 - 



> 



> ^ >0; 
n — 1 

An-1 



rt — 1 



>0; 



A„ > 



An ^ A„_i 

n 71 — 1 



Note that Cn = An-i U {[A„ > 0]}, so by guideline 2, Cn has generating function 

An-l{xi . . . ,Xn-l) 



1 — Xr. 



(12) 
(13) 

(14) 



Since A„ > A„_i is imphed by Vn in H13|l . by guidehne 3, substituting A„ ^ An + An-i in P„ gives 

An-1 



f — 



1 - 2 - 



>^>0; 
n — I 



An > 



n-1 



(15) 



and 



-^n(^li ■ ■ ■ 1 '^^n) — ^ni-^n^ '^n— 1 ^ '^n— I'^n); 



where X„ represents the argument hst xi, . . . ,Xn- Using guidehne 5 again, with c = [A„ > ] 
gives 
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where C„ is (I12|l and where Bn is (lllll . Putting this aU together, we have 

^ni^ji) = Cri(X„) — Dn{Xn) 

Cni^-^n^ Cni^-^rt: -^n—l ^ '^n— l-^n) ^" -^n(^n; ^n— 1 ^ '^n— l^n) 

Substituting from p4|l and Lemma [21 gives the result. □ 

In order to make use of the recurrence of Proposition ^ to prove the generating function H10(l for 
anti-lecture hall compositions, let An{q, s) = An{q^ q,q, . . . ,q,s). Then the recurrence of Proposition 
n becomes 

A„_i(g,g) ^ g(l-g"-ig(S)) 

An(«, s) = q An-i{q,qs) (16) 

l-'S (1 - s)(l - s"gU)) 

with initial condition Ao{q,s) — 1. If we were to proceed as with two- rowed plane partitions, we 
would (i) "guess" the form of v4„(g, s), (ii) prove by induction that it satisfies H16|l . and then (iii) 
show that setting s — q gives (|10() . This would be the easiest proof and it would give a refinement 
of the anti-lecture hall generating function, enumerating solutions according to both the weight and 
the size of the last part: 

Since we have not succeeded in guessing A„(g, s), we follow a different approach. Iterating the 
recurrence of H16|) gives: 



n-1 



Aniq,s) = ^(-l)M„_i_,(g,g),s^<zW ^. (17) 

^ (s;g),+i(l-s"gU)) 



1=0 



Now, setting s = q gives a recurrence independent of s: 



An{q,q) = J2^-lyAn-l-^{q,q)—^ ,„+i^ ■ (18) 

»=0 ('7;9)»+l(l - ^ ^) 

We show by induction that the solution to l|18|l is 

A t \ (~g)" 

An{q,q) = , ■ 
[Q )n 

Assume inductively that An-i-i = {—q)n-i-i/ {q^)n-i-i- Then we need to prove that 

i3n(g)-g("^')Cn(g) _ (-g)n 

l_/r) {q')n' 

with 

c„(g)=x:(-i) vir'\^7 

^ (r)n-l-»(9)»-Hl 

and 

l)/2 { — q)n-l-i 



Bn{q) = 5](-l)^g'(^+^) 



(g^)n-i-j(g)j+i 



11 



We will prove that 



B2n+l{q) — \ 2\'^"'^'^ C2n+l{q) — / 2^ 



(-9) 



2n+l 



{-q)2n q^ ^ I 



C2n{q) 



{~qh 



Therefore, we need to prove the foUowing identities for C„ : 

\i {-q)2n-i {~q)2n+l 



i=0 



(g^)2«-j(g)»+i (g^) 



2n+l 



2n-l 



EM) 



. i-q) 



271 — 1 — 2 



2n 



1 



iq^)2n-l-i{q)i+l {q^)2n {q^)2n 

A few q-series manipulations show that the two previous equations are equivalent to: 

. (-1)" 



Y.{-in-i:q), 

3=0 



n 
3 



(19) 
(20) 

(21) 



Recalling that 



n 

3 



(g-")j-(-l)Jg"J-j(j-i)/^ 



q yil] 

we see that the identity follows from the case a = — l,c— > oo of g-Chu Vandermonde summation 
(1.5.2 in ini), 

{a),{q-%{cq^/ay _ (c/a)„ 



j=0 



{c)j{q)j 



Now we need 



(c)« 

(-'7)2n+l 



j=0 



(g^)2n-i(g)i+l ('7^)2n+l ' 



(22) 

(23) 
(24) 



i=0 



('?^)2n-l-j('?)i+l {q^)2n {q^)2n ' 

The same q-series manipulations show that the two previous equations are equivalent to: 

= (-l)"(z(S) (25) 



Y.{-iy{-i;q), 

j=0 



n 
3 



This follows in a similar way from the "other" g-Chu Vandermonde summation (1.5.3 in |21p. 

{a),{q-%q^ _ a-\c/a)r^ 



E- 



(c)j {q)3 



(c) 



(26) 



under the substitutions a = — 1, c = 0. 



□ 
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6 Lecture Hall Partitions 



In ^j, Bousquet-Melou and Eriksson studied the set of integer sequences A = (Ai, . . . , A„) satisfying 
the constraints 

Ai A2 Ajn 

— > — — >•■•> — > . 

n n — \ 1 

They referred to these as lecture hall partitions and showed that the generating function is 



(27) 



In [2], Andrews showed how to use partition analysis to derive a recurrence for the full generating 
function of Cn- However, substantial new ideas, outside of partition analysis, were required to move 
from this to the solution H27I) . 

In this section, we show that by strategic application of Theorem^ we can derive a recurrence for 
the full generating function of a generalization of that will reduce the proof of (|27|l to a g-series 
calculation (albeit nontrivial). Our derivation here via the five guidelines is both simpler and more 
elementary than the approach in jl7 l (at the expense of a more challenging q-series calculation). 

In J7|, we defined truncated lecture hall partitions to be the integer sequences satisfying: 

Afe 



We showed that if 



Al ^ A2 

n ~ n — 1 



> ...> 



Ai Ao 

— > — — > ...> 



n - k + 1 



n n — 1 n — k + 1 

that is, all parts must be positive, the generating function is 



> 



> 



Ln,k{q) 



n 
k 



2n-fe+l. 



q)k 



(28) 



(29) 



It can be checked that setting k = n and dividing by gives (|27|) . 

Proposition 2 The generating function for truncated lecture hall partitions (|28l) satisfies 

XkLn,k~l{xi, . . . ,Xk-l) Ln,k-l{xi, ■ ■ ■ , Xk~2, Xk^iXk) 



Ln,k{xi, . . . , Xk) 



I - Xk I - Xk 

Zn,kLn,k-l{xi, . . . , Xk~2,Xk-lXk) 
1 Zn,k 



with Zn,k — X1X2 



Proof. Note that Afc_i > Afe is implied by Cn,k, so by guideline 4, CnM = ^nM U {[Afc-i > Afc]}. 
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Now apply guideline 5 with c = [A^-i > "_^^ Afc] to get Cn^k =!>-£■. 

A2 > 



Ai 


> 


A2 


> 


Afe_3 


> 


Afe-2 


> 


Afe_i 


> 


Afe-i 


> 


Afc 


> 



n-fc + 3 '■■-2 



n-fc + 2' 



n-fc + 1 



n — 1 \ 



Afc-3 
Afc-2 



> 
> 



n— fc+4 \ 
„-fc+3'\fc-2 

72 — fc + 3 \ 

r. _t--i-9 -^fc— 1 



Ai 


> 


A2 


> 


Afc-3 


> 


Afc-2 


> 


Afc 


> 


Afc-i 


> 


Afc 


> 



T^A2 



71 — 1 ■ 



n — 


fc+3 


n — 


fc+3 


n — 


fc+2 


n — 


fc + 1 


n — 


fc+2 


Afc 










-^fc-2 

Afc-i 
Afc-1 



(30) 



Afc-i > Afc 
Afc > 

The first system on the right, 2?, implies the constraint \k-\ > 0, so it can be added. Now apply 
guideline 5 to I? using c = [Xk-i > Afc] to get: 



V = 



Ai 


> 






" Ai 


> 


71 \ 




" Ai 


> 


n \ 




A2 


> 






A2 


> 






A2 


> 


n — 1 \ 




fc-3 
fc-2 


> 
> 


n— fc+4 \ 
„-fc+3-^fe-2 

n — fc+3 \ 
n-fc+2'^ 




Afc-3 
Afc-2 


> 
> 


n — k-'rA \ 
n-fc+3 -^^-2 

n — fc+3 \ 
n-fc+2 '^'=-1 




Afc-3 
Afc-2 


> 
> 


n— fc+4 \ 
n-fc + 3 '^-^ 

n— fc+3 \ 
n-fc + 2^'=-l 


■ (31) 


fc-1 


> 


Afc 












Afc 


> 


Afc-1 




fc-1 


> 







Afc-1 


> 







Afc-1 


> 







Afc 


> 







. Afc 


> 







. Afc 


> 








The first system on the right of is just Ln^k~i U {[Afc > 0]}. The second system on the right 
becomes Ln,k-i U {[Afc > 0]} after the substitution Afc ^ Afc + Afc_i. So, by Theorem ^ and 
summarizing so far, we have 

XkLn,k~i{xi, . . . , Xk-i) L„^fc_i(a;i, . . . , Xk~iXk) 



Ln,k{xi, . . . ,Xk) 



-S(xi,...,Xfc), (32) 



(l-Xk) (l-Xk) 

where E{xi, . . . , Xk) is the generating function for the last constraint system, £, in H30() . Apply 
Afc_i ^ Afc_i + Afc to £ followed by Afc ^ Afc + (n — fc + l)Afc-i as illustrated below £ ^ £' T: 

(33) 



Al > ;^A2 



A2 > 

Afc-3 > 

Afc-2 > 

Afc > 



J A3 



n 


-fc- 


h3 


n 


-fc- 


h3 


n 


-fc- 


h2 


n 


-fc- 


hi 


n 


-fc- 


h2 



-Afc-2 

Afc-1 
Afc-1 



Afc-1 > Afc 
Afc > 



Al 
A2 

Afc-3 
Afc-2 



> 
> 



> 
> 



^A2 



n— fc+4 \ 
n-fc + 3 

ii=^(Afc_i+Afe) 



n — fc + 2 

Afc > (n — A: + l)Afc-i 
Afc-1 > 
Afc > 



Al 
A2 

Afc-3 
Afc-2 

Afc 
Afc-1 



> 
> 



;rh--^2 

n — 1 \ 

;7roA3 



n— fc+4 \ 
n-fc+3 Afe-2 

^i^^Afe + (n-fc + 3) Afc- 



— n-fc+2 

> 

> 
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By guideline 3, 



E'{xi, ...Xk)^ F{xi 



E{xi, ...,xk) = . . . , Xfc„2,4-^'a;r'=+\ (34) 

Finally, starting from the last set of constraints in l|33|l . perform the following sequence of 
substitutions 

\i ^ \i + {n-i + l)Afe_i; i = fc - 2, . . . 1, 



as illustrated below: 
Al > 



A2 > ^Aa 



Afe-3 
Afe_2 

Afc 
Afc_i 



> 

> 

> 

> 
Al 

A2 



Afc-a 

Afc_2 



ii-f±|A._2 + (n-fc + 4)A... 



n— fc+3 \ 







> 



> 
> 



A2 > ^Aa + (n - l)Afc_i 



^A2 + n\k- 



n — 1 \ 



^44 A, 



n — fc4-3 \ 



n-fc + 2 

Afc > 
Afc-i > 



Al > 
A2 > 

> 



ifc-3 > 


n — 


fc+4 \ 
fc+3'^ 


ifc-2 > 


n — 
n — 


fc+3 \ 
fc+2 


Afe > 







ifc-1 > 







;r^A2 






ri — 1 \ 







fe-2 
fe 



n — fc+4 ' 



Afc-2 > ^rfJfAfc 

Afc > 
Afc_i > 



(35) 



The resulting system of constraints, G, in (|35|l can be viewed as Cn.k-i, where A^-i has been 
replaced by Afe, together with the constraint [Afe-i > 0]. Thus, 



G{xi, . . . ,Xfe) 



Xk-lLn,k-l{xi, ■ ■ ■ ,Xk-2,Xk) 



(36) 



(1-Xfc_i) 

By guideline 3, the generating function for J- is obtained from G by the sequence of substitutions 



givmg 



F(a:i, . . . ,Xfc) = G{xi, . . . , Xk-2, X1X2 ^ 



n— fc+3 
''fe-2 



a;fc-i, a;fe). 



(37) 



Returning to E in and using and (|S7|l . 
£;(xi, . . . ,a;fe) = F(a;i, 



n-/c+2 n-A;+l n 



■'1 -^2 



^n-fc+1 



fc-2 -^fe-l -^fc 

in,fe-i(a;i, ■ ■ • , a;fe_2, Xfe-iXfc 



n-1 



■ X 



n-fc+1 



(38) 
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Combining (|38|l with (|32|) gives the result. □ 
Let L„_fc(g, s) — Ln,k{q, s). Setting Xk — s and Xi = q for i < k in Proposition [21 gives 

Ln.kil^s) ^ -^—Ln,k-i{q,q) - Ln,k-i{q,sq) ( h ^"^^ ) , (39) 

where z„ t = gn-fc+i 

One would hope to prove H29|l now by finding a closed form 
for Z„_fc(g, s), proving that it satisfies the recurrence and then setting s = g to get (|^ . Since 
we were unable to guess L„^fc(g, s), we proceed as for anti-lecture hall compositions to iterate the 
recurrence ijS^ and get 

_.sq^-^ l_5"-fe+j^(«-fc+j)0--2) + ("+i)-("-,-+^) _ 
Now setting s ~ q we need only a single argument: 

It remains to prove that this recurrence is satisfied by H29|l . We defer the details until a later report; 
our main point was to show that strategic application of the guidelines reduce the truncated lecture 
hall theorem to a g-series computation. 



7 The Five Guidelines Suffice 

Let C be the set of inequalities 

Ci,0 + Ci,!-^! + ^4,2^2 + . . . + Ci,nA„ > 0, l<i<r. (40) 

and let Sq be the set of of nonnegative integer sequences satisfying all constraints in C. In this 
section we show that the five guidelines of Theorem ^ are powerful enough to find the generating 
function of Sq for any integers Qj-. We will assume that all constraints are homogeneous^ i.e., that 
Ci_o = 0. Otherwise, introduce a new variable Aq and let C be the same as C, except that for every 
j, the ith constraint is now: 

Ci,oAo + Ci^iAi + Ci^2^2 + • . . + Ci,riA„ > 0. 

Then Fc{xi, . . . Xn) is the coefficient of xq in Fc'{xo, xi, . . . Xn)- We also generalize the claim a bit 
to allow any of the constraints of C to be equalities. 

Theorem 2 The five guidelines of Theorem^ are sufficient to find the full generating function for 
any homogeneous system of linear inequalities and equalities. 

Proof. Let C be a homogeneous system of linear inequalities and equalities with variables Ai, . . . , A„. 
Since we require nonnegative integer solutions, we can assume that for each variable Ai, C contains 
a constraint bi of the form [A.; > 0] or [A.; = 0] . Call these constraints bi basic. Write C in the form 

C = [Ci, C2, . . . , Cr] fol, 62, ■ • • , bn], 
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where ci, C2, ■ ■ ■ , Cr is an ordered list of the non-basic constraints in C If r = 0, aU constraints are 
basic and the generating function foUows from guidchnes 1 and 2 (and Fi^Xi=o]{^i) = !)• 

Otherwise, define: 

M: the largest positive coefficient of ci (0, if none); 

&max'- the number of occurrences of Af among the coefficients of ci; 

m: the smallest negative coefficient of ci (0, if none); 

erain'- thc numbcr of occurrences of m among the coefficients of ci. 

When r > we show that we can use the guidelines to reduce the computation of the generating 
function of C to the computation of the generating function of one or more systems C in which at 
least one of the statistics {r, M, Cmax, |™|, ernin} has been reduced. 

If m — 0, all coefficients of Ci are nonnegative, so Ci is redundant and can be deleted. Otherwise, 
if M = 0, all coefficients of Ci are nonpositive and so we get an equivalient system replacing Xj by 
in ci, . . . , Cr and setting bj = [Xj =0]. In so doing we have decreased |m| or emin- 

Otherwise, m < and M > 0; we do a version of Elliott reduction |20| . Let i and j be such that 
m is the coefficient of Xi in ci and M is the coefficient of Xj. We would like to use guideline 3 and 
reduce to a system with smaller M or Cmax or Itt^I or Grnin- First use guideline 4 with c — [A^ ^ 

Fc{Xn) = Fcu[\i>\j]{Xn) + i^CU[Aj>A,](-'^rl)■ 
For the first term, ^cu[Ai>A -]{Xn), do the substitution Xi <— Xi+Xj into constraints ci, C2, . . . , in C. 
This decreases the coefficient of Xj, thereby decreasing M or Cmax- By guideline 3, the substitution 
Xj -i— XjXi in the generating function of the resulting constraint system gives -Fcu[Ai>Aj] (^n)- 

For the second term, -Fcu[Aj>Ai] (-'^n)) if = i^j = 0], there are no solutions and the generating 
function is 0. Otherwise, bj — [Xj > 0]. Substitute Xj <— Xj +Xi into constraints Ci, C2, . . . , in C to 
get C. This increases the coefficient of A^, thereby decreasing |m| or Cmin- Substituting Xj ^ Xj+Xi 
into [Xj > Xi] gives [Xj > 0]. By guideline 3, 

Fcul\j>X^]{Xn) = Fc'UlXj>0]{Xn;Xt ^ XiXj). 

However, we disallow strict inequalities. So, use guideline 5 with c — [Xj > 0] and observe that 
bj = [Aj > 0] e C. Let C" denote C with b^ [A^ 0]. Then 

Fc'yj[\j>0\{X-n) = Fc'{Xn) — Fc'yj[\^<Q]{Xn) = Fc'(X„) — Fc"(X„). 

□ 

(Note that we have optimized the proof for simplicity at the expense of algorithmic efficiency.) 

It follows from Theorem |21 and its proof that the full generating function of (|40|l can be built up 
from the functions 1/(1 — a;^) by a finite number of additions, subtractions, and substitutions. We get 
then as a corollary the following well-known result: The full generating function for the nonnegative 
integer solutions to any system of linear inequalities in n variables with integer coefficients has the 
form 

p(xi, . . . ,x„) 

(l-ai)(l-a2)---(l-at)' 
where i > 0, p is a polynomial in xi, . . . , a;„ and each ai is a monomial in xi, . . . , a;„. 
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8 Relationship to MacMahon's Partition Analysis 



We give a brief introduction to partition analysis in order to highlight the fact that the guidelines of 
Theorem n underlie the work of MacMahon. Indeed, they were distilled from partition analysis by 
a study of MacMahon's work in [53] and its application by Andrews, Paule and Riese in the series 

of papers [3 13 H US □ El El HI HH . 

Consider the set of constraints C = {ci, . . . c^} where 

Ci = [aj,iAi + • • • + ai,„A„ > 0]. 

We seek the full generating function for the set Sc of nonnegative integer sequences A = (Ai, . . . , A„) 
satisfying Ci, 1 < i < r: 

Fci^l 1 ■ ■ ■ 1 ^n) — ^ ^ ' ' ' -^Ti" ■ 

AeSc 

The method of partition analysis, developed by MacMahon in j25| is to view the problem as follows. 
Let 

Expanding P gives 

P{xi,X2,-.-,Xn,Ci,C2,...,Cn) = 



Observe that A S 5'c iff in the term corresponding to A in the sum 142|) every Ci, 1 < i < r, 
has nonnegative exponent. Thus Fc{xi, . . . , Xn) is recovered from P{xi, X2, ■ ■ ■ , Xn, ci, C2, . . . , c„) by 
deleting all terms in which some Ci has a negative exponent and then setting ci = C2 = • ■ • = = 1. 
MacMahon uses the Omega operator to express this process: 

Fc{xi, . . . ,Xn) = ^ P{X1,X2, ■ ■ ■ ,X„,Cl,C2, . . . ,C„). 

The core of partition analysis is a system of Omega-rules designed to be applied strategically to 
transform P(xi, X2, ■ ■ ■ , Xn, ci, C2, . . . , c„) step-by-step into Fc(xi, . . . , a;„). This view converts the 
combinatorial problem into an algebraic one, opening the possibility, for example, of a partial fraction 
decomposition of H41|) to assist in the transformation from P to F. A list of basic Omega-rules 
appears in |23(pp. 103-106) and (5]. 

This approach has proven both powerful and systematic in the computer solution of systems of 
inequalities. However, for deriving recurrences for infinite families, we found that a return to some 
of the basic underlying ideas simplified the process. We note the roots of guidelines 3-5 of Theorem 
12 in the work of MacMahon j^Sj- 

Our use of guideline 3 (which performs limited column operations on the constraint matrix) is 
used to much the same effect as the following Omega-rule: 



I ■ • ■ I "'n 



)=nT 



(41) 



Ai,...A„>Oj = l 



^2 ' ' ' 1 1 

Ai,...A„>0 \ i=l 



(42) 



g (l-x,c)(l-ff) {l~x,){l-x,xty 
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The utility of guideline 4 was recognized by MacMahon. He writes in P- 103, "A very useful 
principle is that of adding an inequality which is afortiori true." It is also used in a decomposition 
shown at the beginning of 25 , Section 379, p. 131. Guideline 5 is one of MacMahon's Omega-rules, 
found in [221, Section 351, p. 104 (slightly transformed): 

n Pic) = F(i)- n Pii/c). 
> > 

9 Concluding Remarks 

The "five guidelines" of Theorem 1 provide a unified setting for computing the full generating 
function for many challenging families of constraints. However, even though they are guaranteed to 
be sufficient to find the generating function for any homogeneous linear system, we are not necessarily 
guaranteed to be able to use them to devise a recurrence for a parametrized family of constraint 
sets. 

In continuing work we consider the case when all constraints have the form > \j or > Aj , 
forming a directed graph. We show how to get a recurrence by strategically manipulating the 
diagrams. Many examples are presented, including two- and three-rowed plane partitions, plane 
partitions with diagonals, plane partition diamonds, and hexagonal plane partitions. 

Finally, we note that in Xin offers a speed-up to the Omega package for implementing 
MacMahon's partition analysis. Xin's method uses the theory of iterated Laurent series and partial 
fraction decompositions. 

Acknowledgement. We are grateful to the referee for a careful reading of the manuscript and 
detailed suggestions to improve the presentation. 
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